(12) INTERNATIONAL APPLICATION PUBLISHED UNDER THE PATENT COOPERATION TREATY (PCT) 



(1?) World InleDectual Property 
Oi^aiuzation 
Internationa] Bureau 

(43) International Publication Date 
24 June 2004 (24.06.2004) 




(10) International Publication Number 

PCT WO 2004/053829 Al 



(51) International Patent Classification'': G09G 5/00 

(21) International Application Number: 

PCTAJS2003A)38862 

(22) In tcmaUonal Filing Date: 8 December 2003 (08. 1 2.2003) 

(25) Filing Language: English 

(26) PubUcation Language: English 



(30) Priority Data: 
60/431,662 



8 December 2002 (08.12.2002) US 



(71) Applicant (for all designated States except US): IMMER- 
SION CORPORATION [USAJS]; 801 Fox Lane, San 
Jose, CA 95131 (US). 

(72) Inventors; and 

(75) Inventors/Applicants (for US only): GRANT, Danny 
[CA/CA]; 6678 Chateaubriand, Montreal, Quebec H2S 
'>N7 (CA). EID, Jeffrey [US/US]; 105 Shadwell Drive, 
Danville, CA 94506 (US). ENDO, Sholchi [JP/US]; 20268 
Northwest Square, Cupertino, CA 95014 (US). 



(74) Agents: ALEMANNI, John, C. et al.; Kilpatrick Stock- 
ton LLP, 1001 West Fourth St, Winston-Salem, NC 27101 
(US). 

(81) Designated States (national): AE, AG, AL, AM, AT, AU. 
AZ, BA, BB, BG, BR, BY, BZ, CA, CH, CN, CO, CR, CV, 
r7.> DE, DK, DM, DZ, EC. EE, EG, ES, FI, GB, GD, GE, 
GH, GM, HR, HU, ID, IL, IN, IS, IP, KE, KG, KP, KR, 
KZ, LC, LK, LR, LS, LT, LU, LV, MA, MD, MG, MK, 
MN, MW, MX, MZ, NI, NO, NZ, OM, PG, PH, PL, PT. 
RO, RU, SC, SD, SE, SG, SK, SL, SY, TJ, TM, TN, TR, 
rr, TZ, UA, UG, US, UZ, VC, VN, YU, ZA, ZM, ZW. 

(84) Designated States (regional): ARIPO patent (BW, GH. 
GM, KE, LS, MW, MZ, SD, SL, SZ, TZ, UG, ZM, ZW), 
Eurasian patent (AM, AZ, BY, KG, KZ, MD, RU, TJ, TM), 
European patent (AT, BE, BG, CH, CY, CZ, DE. DK, EE, 
ES. n, FR. GB, GR, HU, IE, IT. LU, MC, NL, PT, RO, SE, 
SI, SK. TR), OAPI patent (BF, BJ, CF, CG, CI, CM, GA, 
GN, CK3, GW, ML, MR, NE, SN, TD, TG). 



Published: 

— with imemaiional search report 



[Continued on next page] 



(54) Title: MEITIODS AND SYSTEMS FOR PROVIDING A VIRTUAL TOUCH HAPTIC EFFECT TO HANDHELD COM- 
MUNICATION DEVICES 



100 



00 

in 



JJO 

A 



J50 



195 



240' 



Memory 




Transceiver 







Operating 
System 






i 




API 



Processor 



"7" 

120 



290 



Actuator 



Display 



260 



270 



Audio 



7 

130 



m 



User-Inter&ce 



112 



O (57) Abstract: Embodiments of the invention relate to methods and systems (100) for providing customized "haptic messaging to 
® use of handheld communication devices in a variety of applications. In one embodiment, a method of providing virtual touch to a 
^ handheld communication device includes: receiving an input signal associated with a virtual touch; outputing a request relating to a 
O contact with a user-interface member coupled to a handheld communication device; and providing a control signal associated with 
^ the contact to an actuator coupled to the handheld communication device, the control signal being configured to cause the acwator 
to output a haptic effect associated with the virtual -touch. 



METHODS AND SYSTEMS FOR PROVIDING A VIRTUAL TOUCH HAPTIC 
EFFECT TO HANDHELD COMMUNICATION DEVICES 

INVENTORS 
Danny Grant, Shoichi Endo, and Jeffrey Eid 
CROSS-REFERENCE TO RELATED APPLICATIONS 
This application claims priority to U.S. Provisional Patent Application No. 
60/431,662, filed on December 8, 2002, the entire disclosure of which is incorporated herein 
by reference. 

FIELD OF THE INVENTION 

This invention relates generally to haptic-feedback systems. More specifically, 
embodiments of tlie present mvention relate to using customized haptic effects in a variety of 
applications to convey information to users of handheld comnaxmication devices. 

BACKGROUND 

As handheld communication devices become part of everyday life, device 
manufactures and service providers strive to enhance the versatility and performance of such 
devices. 

Handheld communication devices in the art (e.g., mobile phones, pagers, personal 
digital assistants (PDAs), etc.) typically use auditory and visual cues, to alert a user when 
incoming messages, such as voice calls and emails, are received. Such auditory and visual 
alerts, however, have the disadvantages of being distracting in some situations (e.g., during 
driving), or annoying in others (e.g., during a meeting or a concert). Although vibratory 
alCTts are made available in some communication devices subh as cellular phones, such 
vibratory effects cannot be customized or personalized according to ^plications, thus 
conveying little information to the user. A need, therefore, exists in the art for a new sensory 
modality that delivers information to users of handheld communication devices in a 
personalized fashion. 

SUMMARY 

Embodiments of the invention relate to methods and systems for providing 
customized "haptic messaging" to users of handheld, communication devices in a variety of 
applications. 

In one embodimoit, a method of providing virtual touch to a handheld 
communication device includes: receiving an input signal associated witii a virtual touch; 
outputting a request relating to a contact with a user-interface member coupled to a handheld 
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communication device; and providing a control signal associated with the contact to an 
actuator coupled to the handheld communication device, the control signal being configured 
to cause the actuator to output a haptic effect associated with the virtual touch. 

In another embodiment, a method of providing virtual touch to a handheld 
5 communication device includes: receiving a virtual touch indicator; performing an 
initialization responsive to the virtual touch indicator on a handheld communication device; 
receiving a virtual touch signal associated with the initialization; and outputting a control 
signal associated with tiie virtual touch signal to an actuator coupled to the handheld 
communication device. 

10 Further details and advantages of embodiments of the invention are set forth below. 

BRIEF DESCRIPTION OF THE FIGURES 
These and other features, aspects, and advantages of the present invention are better 
imderstood when the following Detailed Description is read with reference to tlie 
accompanying drawings, wherein: 
15 FIG. 1 depicts a block diagram of a haptic handheld communication device according 

to an embodiment of the present invention; 

FIG. 2 shows a flowchart depicting a method of using customized haptic effects to 
convey information to users of handheld communication devices, according to an 
embodiment of the invention; 
20 FIG. 3 shows a flowchart depicting a method of using haptic logos to relate 

information to users of handheld communicatidn devices, according to an embodiment of tlie 
invCTtion; 

FIG. 4 shows a flowchart depicting a method of haptically encoding communication 
signals, according to an embodiment of the invention; 
25 FIG. 5 shows a flowchart depicting a method of providing haptic messaging to users 

of handlield communication devices, according to a further embodiment of the invention; 

FIG. 6 shows a flowchart illustrating a method of providing an interactive virtual 
touch in one embodiment of the present invention; 

FIG. 7 depicts a flowchart illustrating a method of carrying out a chat session using 
30 handheld conmiunication devices, according to an embodiment of the invention; 

FIG. 8 shows a flowchart depicting a nietiiod of using haptic effects to relate 
navigation information, according to an embodiment of the invention; and 
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FIG. 9 shows a flowchart iUustrating a method for providing haptic effects to a remote 
control in one embodiment of the present invention. 

DETAILED DESCRIPTION 
Embodiments described in the following description are provided by way of example 
5 to illustrate some general principles of the mvention. and should not be constraed as limiting 
the scope of the invention in any manner. One skilled in the art would also recognize that 
various changes and modifications can be made herein, without departing fiom the principles 
and scope of the invention. 

FIG. 1 depicts a block diagram of a handheld communication device 100 according to 
10 an embodiment of the invention. It will be appreciated that various elements are shown in 
schematic form for illustrative purposes and are not drawn to scale. It will also be 
appreciated that many altemative ways of practicing the present invention exit. Accordingly, 
various changes and modifications may be made herein, without departing fi-om the 
principles and scope of the invention. 
15 Device 100 includes a device body including a housing 110 and a user-interface 112; 

a processor 120; at least one actuator 130 in'' communication with processor 120; and a 
memory 140 in communication with processor 120. Device 100 also includes an antemia 150 
and a transceiver 160, in communication with processor 120. Device 100 additionally 
includes a display module 170 and an audio module 180, in communication with processor 
20 120. Display module 170 may include, for example, a liquid crystal device. Audio means 
180 may include, for example, a speaker, a microphone, and the like. 

For purpose of illustration in the embodiment of FIG. 1, processor 120, actuator 130, 
and memory 140 are shown to be enclosed within and coupled to the device body. Such an 
illustration, however, should not be construed as limiting the scope of the invention in any 
25 manner, hi altemative embodiments, actuator 130 may, for example, be coupled to the 
outside of housing 110, or embedded in housing' 110 via a suitable mechanism. Furtlier, user- 
interface 112 may include one or more user-interface members. As used herein, a user- 
interface member includes, witliout limitation, • a key pad having one or more keys, one or 
more buttons, a touch screen or touch pad, a scroll wheel, a direction pad, a trackball, a knob, 
30 a miniature joysticlc, or other user-interface means Icnown in the art. 

Device 100 further includes an API (AppUcation Program hiterface) 190, working in 
conjunction with an operating system 195. A device driver (not shown) may optionally 
provide an interface between operating system 195 and processor 120. 
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Memory 140 of device 100 stores a program code that includes instructions to cause 
processor 120 to perform various tasks. The following description provides some examples. 

FIG. 2 shows a flowchart 200 depicting a method of using customized haptic effects 
to convey information to users of handheld communication devices, according to an 
5 embodiment of tlie invention. At step 210, an input signal associated with an event is 
received. At step 220, a source of the event is determined and a control signal is selected 
based on the determination. At step 230, a control signal is output to an actuator coupled to a 
handheld commimication device (see FIG. 1 for an embodiment of such device). The control 
signal is configured to cause the actuator to output a haptic effect associated wifh the event. 

10 Furthermore at step 240, a collection of haptic effects is provided, each haptic effect 

being associated with a control signal. For example, memory 140 of FIG.. 1 can store a 
program code that includes instractions to generate the control signals (e.g., each 
characterized by a distinct waveform) for rendering the corresponding haptic effects. Haptic 
effects (along with associated control signals) may also be downloaded or transmitted from a 

15 remote source, such as a service provider, a network resource, a Web server, a remote 
handheld communication device or computer. Such downloaded or transmitted haptic effects 
can be further edited or modified. At step 250, a mapping between an event of interest and 
one of the stored haptic effects is received. By way of example, memory 140 of FIG. 1 may 
also store a program code that enables a user to map an event of interest to one of the haptic 

20 effects as provided, e.g., via user-interface 112 througli API 190, where the event may be 
identified by its source. At step 260, the one-to-one mappings made between various events 
of interest and the corresponding haptic effects are compiled into a haptic lookup table, which 
can, for example, be stored in memory 140 of FIG. 1. 

In the embodiment of FIG. 2, the term "selecting" includes, witliout limitation, 

25 looking up a predetermined mapping between the event of interest and a corresponding haptic 
effect based on the source determination, and selecting/generating a control signal that is 
configured to render the desired haptic effect associated with the event (e.g., upon being 
applied to an actuator). Selection can be made based upon the aforementioned haptic lookup 
table, for example. 

30 In one embodiment, the iuput signal may include a communication signal associated 

with a call event, such as a voice call, an e-mail, or a message in text or multimedia form, 
which may be received via antenna 150 and transceiver 160 of FIG. 1, for example. Tlie 
"source" of a call event may be related to a characteristic that distinctly identifies or 
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characterizes the call event, such as the caller's phone number, the sender's e-mail address, a 
graphical feature or an icon associated with the incoming message, etc. 

In another embodiment, the input signal may be associated with a reminder event, 
which may be a self-generated message on the handheld communication device serving as a 
5 reminder for a pre-scheduled activity (e.g., an appointment or a meeting). The source in this 
scenario may be associated with the type of a pre-scheduled activity (e.g., a business meeting 
vs. a restaurant reservation), or the time at which the pre-scheduled activity takes place. 

In yet another embodiment, the input signal may include a conmiunication signal 
associated with a status eyent, for example, received via anteima 150 and transceiver 160 of 
10 FIG. 1. Examples of a status event include, but are not limited to: an advertisement (e.g., 
sale) event, a one-to-one marketing event, a business-transaction event, a stock-trading event, 
a weather-forecast event, a sports (or game) event, an entertainment event, and an emerg^icy 
(e.g., 911) event In this scenario, the source may be associated with a characteristic tiiat 
distinctly identifies the sender and/or the nature of a status event, such as the phone number 
15 of the handheld user's stock broker, the e-mail address of the user's favorite store, the logo 
associated with the user's favorite TV or radio station, and so on. 

In one embodiment, an event of interest can be accompanied by a distinct haptic 
effect, or overlapping haptic effects, conveying to the user customized infonnation such as 
"who is calling," "what is happening," and so on. The user can also be allowed to update the 
20 haptic lookup table, e.g., to include new events, and/or to modify the mappings between the 
existing events of interest and the corresponding' haptic effects. 

Moreover, a specific haptic effect can be assigned to any incoming signal event whose 
source is vmknown, so as to alert the user tiiat the incoming message is firom an un- 
identifiable or sender, 

25 As used herein, the term "handheld conmixmication device" includes, without 

limitation, a mobile phone such as a cellular phone or a satellite phone, a personal digital 
jassistant (PDA), a cordless telephone, a pager, a two-way radio, a handheld or portable 
computer, a game console controller, a personal gaming device, an MPS player, or otiier 
personal electronic devices known in the art that are equipped with communication or 

30 networking capabilities. 

In one embodiment, the aforementioned haptic effects can be used as haptic ringers 
(e.g., counterparts to auditory ring tones) that are customized or personalized to convey 
infonnation to tlie user about various events of interest By way of example, a haptic ringer 
associated with a call firom a loved one (e.g., the user's spouse) may comprise low-amplitude 
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aiid high frequency vibrations that impart gentle sensations to the user. Li contrast, a haptic 
ringer associated with an emergency event (such as a 911-call) may comprise jolt-like pulses 
that impart pounding sensations to the user. 

In contrast with conventional auditory ring tones, the aforementioned haptic effects 
5 (e.g., haptic ringers) are more desirable in an environment where extraneous auditory signals 
are prohibited (e.g., during a meeting or a concert), and/or where it is difficult to distinguish 
auditory signals (e.g., in a loud environment such as an airport). The haptic ringers are also 
more suitable in- distracting situations such as driving, so that the user of a handheld 
communication device can keep eyes on the road without having to look at the device. 
10 Moreover, such haptic ringers convey customized infomiation to the user, so that the user is 
aware of *Svho is calling," '"what is happening," and so on, as the following examples further 
illustrate. 

A handheld communication device such as a mobile phone may be configured to 
allow a user to include haptic infonnation or a haptic code in an outgoing communication 

15 signal, e.g., carrying a voice call, an e-niail, or a message. The encoding of a CQmmunication 
signal with haptic infomiation may be based on an estabhshed scheme or protocol, and/or on 
a per-system basis. The haptic code is configured to cause a haptic effect to be output when 
the commimication signal is delivered to another handheld communication device. In one 
embodiment, businesses and organizations may each be associated with a distinct haptic logo 

20 (e.g., a paiticular vibration pattern) and include their haptic logos in various messages sent to 
the handheld communication devices of their customers. Such haptic logos can serve as 
counterparts to conventional logos known in the art, for example. Various status events 
mentioned above may also be transmitted in this maimer. By way of example, a merchant 
may include its haptic logo in various advertisement events and business transaction events to 

25 be transmitted to the handheld communication devices of its customers. Stock brokers (or 
brokerage firms), TV or radio stations, and marketing/advertising agencies may likewise 
include their haptic logos in various stock-trading events, weather-foreciast events, sports 
events, entertainment events, and one-to-one marketing events to be transmitted to the 
handheld users. 

30 FIG. 3 shows a flowchart 300 depicting a method of using haptic logos to relate 

infonnation to users of handheld commmoication devices, according to an embodiment of the 
invention. A handheld communication device receives an input signal at step 310, the input 
signal being associated with a status event. The handheld communication device extracts a 
haptic code fi-om the input signal at step 320,' where the haptic code is associated with a 
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haptic logo. At step 330, the handheld communication device provides a haptic effect 
associated with the haptic logo. . Step 330 may include providing a control Signal to an 
actuator coupled to the handheld commxmication device, v^here the control signal is based at 
least in part on the haptic code and configured to cause the actuator to output the haptic 
5 effect. 

In one embodiment, the extracted haptic code may be dii'ectly applied to the actuator 
for rendering the desired haptic effect. In another embodiment, the haptic code may be 
configured according to a predetermined scheme or protocol that includes, for example, a 
table of haptic codes (some of which may be associated with one or more haptic logos) versus 

10 control signals for rendering the corresponding haptic effects. In this way, a processor in the 
handheld communication device can look up the corresponding control signal from the table 
based on the extracted haptic code, and output the selected control signal to the actuator for 
rendering the desired haptic effect 

In the embodiments of FIG. 2 or 3, the handheld communication device (or the haptic 

15 code) may be programmed such that the haptic eflfect is output immediately, or at a 
prescribed time after receiving the input signal, as desired in applications. The haptic effects 
can also be triggered by, or synchronized with, other occurrences. 

A handheld commimication device may be further configured such that some of its 
user-interface members (such as those described above) are each associated witli a haptic 

20 code, e.g., according to a predetermined scheme or protocol. In one embodiment, some of 
tliese haptic codes may be associated with haptic effects that emulate expressions or 
behaviors, such as "laugh," "giggle," "hug," "high-five," "heartbeat," "pet purring," etc. This 
allows haptic effects to be transmitted and experienced, e.g., in an interactive conversation or 
a chat session, by pressing or manipulating such members. 

25 By way of example, suppose that user A (termed "Alice" herein) is engaged in a chat 

session with user B (temied "Bob" herein) Via their respective mobile phones. In one 
embodiment, when Bob tells Alice a joke, Alice can respond by sending a "laugh" sensation 
to Bob, e.g., by pressing a key on her mobile phone that is assigned with a haptic code 
corresponding to a laugh sensation. This causes a signal to be ti'ansmitted from Alice's phone 

30 to Bob's phone, and a corresponding haptic effect to be output to Bob's phone (and thereby 
experienced by Bob). In alternative embodiments, Alice can include a haptic code in an 
outgoing message (which may also contain a video image such as a picture taken by her 
mobile phone, and/or a giaphical featm-e such as an emoticon emulating a smiley face) to be 
transmitted to Bob, e.g., by pressing the corresponding user-interface member. The haptic 
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code causes a haptic efiFect to be output when the message is delivered to a remote device 
such as Bob's mobile phone. In one embodiment, the haptic effect may be correlated or 
synchronized with the displaying of a video image contained in the message. In another 
embodiment, the generation of the haptic effect based on the haptic code maybe carried out 
5 in a manner similar to that described above with respect to the embodiment of FIG. 3 . 

FIG. 4 depicts a flowchart 400 illustrating a method of a method of haptically 
encoding communication signals, according to an embodiment of the invention. At step 410, 
an input signal associated with an actuation of a user-interface member is received. By way 
of example, the input signal- may be associated with Alice's pressing or manipulating a 
10 particular user-interface member. At step 420, a h^tic code associated with the actuation is 
determined. At step 430, the haptic code is included in an output signal, and the output signal 
is sent to a remote handheld communication device. As described above, the output signal 
may also include a message, a video image, and'br a graphical feature. 

A handlield communication device may also be configured such that a haptic effect, 
15 along with a message, is output upon a contact with a user-interface member being made 
(e.g., by a user or an input device). FIG. 5 depicts a flowchart 500 illustrating a method of 
haptic message that can be associated with this situation, according to an embodunent of the 
invention. ' At step 510 of the flowchart 500, a handheld communication device receives an 
input signal. At step 520, the handheld communication device outputs a reqtxest for a contact 
20 with a user-interface member coupled to the handheld communication device. At step 530, 
the handheld communication device provides "a control signal associated with the contact to 
an actuator coupled to the handheld communication device. The control signal is configured 
to cause the actuator to output a haptic effect associated with the input signal. Step 520 may 
include having a visual effect displayed, an auditory effect played, and/or a distinctive haptic 
25 ringer output, which requests a contadt with the user-mterface member being made. 

In one embodiment, the input signal in FIG. 5 may include a haptic code, along with a 
message, a video image, and/or a graphical feature, etc. For example, the haptic code may be 
configured to cause a "hug" sensation to be output when the video image contained in the 
input signal is displayed. The input signal may also contain a provision or protocol that 
30 specifies that the incoming message along with the corresponding haptic effect is output upon 
a contact vdth a particular user-interface member (e.g., the #5 key) being made. 
Alternatively, the handheld communication device may detamine the user-interface member 
to be contacted, before outputtmg incoming message along with the corresponding haptic 
effect. 
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In another embodiment, the input signal of FIG. 5 may be associated with a "virtual 
touch," e.g., to mimic a handshake, a "high-five," a pat on the back, a pulse or heartbeat 
sensation, a pet purring sensation, or other touch sensations associated with human (and/or 
human-animal) interactions. In one scenario, the input signal at step 510 may include a 
"virtual touch indicator," based on which the request for a contact with a particular user- 
interface member is made. The virtual touch indicator may be in the form of a haptic code, a 
message, or other informative means. The control signal at step 530 may be generated, e.g., 
based on the virtual touch indicator, a haptic code associated with tlie user-interface member 
at play, or other predetermined scheme. The input signal at step 510 may also include a 
virtual touch indicator along with a virtual touch signal for rendering the desired haptic 
effect In this case, the control signal at step 530 may be based on the virtual touch signal. 

Referring back to the chat session between Alice and Bob, by way of example at the 
end of their chat session, Alice may wish to send Bob a "liigh-five." She sends to Bob's 
mobile phone a signal including a virtual touch indicator, which in turn prompts a request that 
Bob be in contact with a user-interface member coupled to liis phone, such as a direction pad 
(e.g., by putting his fingers on the individual keys of the direction pad), a key pad, a touch 
screen, a trackball, a joystick, or the like. The control signal for rendering a haptic effect that 
emulates a "high-five" may be based on the haptic code associated with the user-interface 
member, transmitted with the input signal from Alice, and/or other predetermined scheme. 

Interactive virtual touch can also be engaged between users of handheld 
communication devices, where the manipulation of a user-interface member on one handheld 
communication device is transmitted possibly in substantially real-time to another handheld 
device and experienced by its user, and vice versa. FIG. 6 depicts a flowchart 600 illustrating 
a method of providing interactive virtual touch in one embodiment of the present invention. 
In the embodiment shown, a handheld conmiunication device first receives an input signal 
including a virtual touch indicator at step 610/ A distinctive haptic ringer may, for example, 
accompany the arrival of the virtual touch indicator, identifying the sender and the nature of 
the input signal. The handheld communication device may then perfonn any necessary 
initialization to enable the conamunication at step 620, which may also include requesting a 
contact with a particular user-interface member coupled to the handheld conmiunication 
device at step 625. The handheld communication device subsequently receives a virtual 
touch signal in the commimication associated with the desired haptic effect at step 630. The 
handheld communication device provides the haptic effect at step 640, e.g., by applying the 
virtual touch signal to an actuator coupled to the user-interface member. 
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In one embodiment, the virtual tench signal may be associated with the manipulation 
of a user-interface member on a remote handheld device and transmitted in substantially real- 
time. And the user on the receiving end may respond by acting in a similar fashion, so as to 
emulate an interactive touch. Any schemes for. delivering virtual touch to users of handheld 
5 communication devices may be used. 

Haptic effects can also be used to enhance and complement the information content 
communicated between handheld conraiunication devices. In one embodiment, a plurality of 
handheld communication users may be engaged in a chat session via their handheld 
communication devices. The users may each have a graphical representation or avatar 

10 displayed on otlicar handheld conmiunication devices. Such avatars can also be haptically 
enabled, for example, whereby their expressions and/or behaviors are accompanied and 
enhanced by corresponding haptic effects. FIG. 7 is a flowchart 700 depicting a method of 
carrying out a chat session using handheld comtnimication devices, according to an 
embodiment of tlie invention. In the embodiment shown, a handheld conmiunication device 

15 receives an input signal associated witli a that message at step 710. The handlield 
communication device displays an avatar associated with the chat message at step 720. The 
avatar may be shown on display 170 of FIG. 1, in one embodiment. At step 730, the 
handheld conmiunication device provides a haptic effect associated with the chat message. 
Step 730 may include outputting a control signal to an actuator coupled to tlie handheld 

20 communication device, where the control signal is configured to caiise the actuator to output 
the haptic effect. In one embodiment, the haptic effect may be correlated with an expresision 
or behavior of the avatar, such as a laugh or giggle, a cry, a pet purring, or the Uke. 

Handheld communication devices are increasingly equipped with navigation 
capability, for example, in communication v^th the Global Position System (GPS) or other 

25 navigation systems. Haptic effects can also be used to convey navigation infomiation, such 
as positional and/or directional information, to handheld users. By way of example, FIG. 8 
shows a flowchart 800 depicting a metliod of haptic navigation, according to an embodiment 
of the present invention. Tlie flowchart 800 discloses receiving an input signal associated 
with a position of a handlield communication device at step 810; deteiTnining the position of 

30 a handheld communication device relative to a predetermined location at step 820; and 
providing a haptic effect associated wdth the determination at step 830. Step 830 may include 
outputting a control signal associated with the determination to an actuator coupled to the 
handheld communication device, the control signal being configured to cause the actuator to 
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output the haptic effect. Further, the input signal at step 810 may be received from GPS, a . 
digital compass, or other navigation systems known in the art. 

In one embodiment, the haptic effect may be associated with a distance between the 
position of the handheld commxmication device and a predetermined location (termed 
5 "destination" herein). For example, the haptic effect may include a vibration having a 
magnitude and a frequency, where at least one of the magnitude and the frequency decreases 
as the distance from the destination diminishes. Additionally, the haptic effect may be 
configured to convey a quantitative measure of the distance. By way of example, tlie haptic 
effect may include one or more pulse or jolt sensations, where the numbn of pulses is 
10 proportional to tlie nimiber of miles between the position of tlie handheld device and the 
destination. 

Processors described above (including processor 120 of FIG. 1) can include, for 
example, one or more digital logical processors capable of processing input, execute 
algorithms, and generate output as necessary to perform various tasks, such as those 

15 described above. Such processors/controllers may include a microprocessor, an Application 
Specific Integrated Circuit (ASIC), and state machines. Such processors include, or may be 
in communication with, media (including memory 140 of FIG. 1). Such media include, for 
example, computer readable media, which stores program code that, when executed by a 
processor, cause the processor to perform the steps described herein. Embodiments of 

20 computer-readable media include, but are not hmited to, an electronic, optical, magnetic, or 
other storage or transmission device capable of providing a processor, such as tlie processor 
in a web server, witli computer-readable instructions. Other examples of suitable media 
include, but are not limited to, a floppy disk, CD-ROM, magnetic disk, memory chip, ROM, 
RAM, ASIC, configured processor, all optical media, all magnetic tape or otlier magnetic 

25 media, or any other medium fi-om which a compute processor can read. Also, various other 
forais of computer-readable media may transmit or carry instructions to a computer, 
including a router, private or public network, or other transmission device or channel. 

Program code and associated application programs related to various applications may 
also reside on a remote source, such as a network resource, a Web server, a rejnote handheld 

30 communication device or computer, which can be tiansmitted or downloaded to a handheld 
commimication device on a regular or predetermined basis. Haptic effects (along with 
associated control signals) can also be downloaded or transmitted from a remote source, as 
described above. 
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Actuators described above (including actuator 130 shown in FIG. 1) can include, for 
example, a pager motor, an eccentric rotating motor, a harmonic eccentric rotating motor, a 
voice coil, a solenoid, a resistive actuator, a piezoelectric actuator, an electro-active polymer 
actuator, or other types of active/passive actuators suitable for generating haptic effects. U.S. 
5 Patent Nos. 6,429,846 and 6,424,333 disclose further details relating to some of these 
actuators, both of which are incorporated in full herein by reference. In some embodiments, 
one or more actuators may be implemented in a handheld communication device, configured 
to deliver appropriate haptic effects. It will be appreciated that various control schemes can 
be devised accordingly, for controlling the actuator(s) in a manner that best acliieves the 

10 desired haptic effects. 

Referring back to FIG. 1. In one embodiment, actuator 130 may be coupled to 
housing 110, thereby imparting haptic eflFects thus generated to the device body. Haptic 
ringers (or alerts) described above may be delivered in this manner, for instance. In another 
embodiment, actuator 130 may be coupled to user-interface 112 of tlie device body. For 

15 instance, an active and/or resistive actuator can be coupled to user-interface 112 to deliver a 
virtual touch described above. One or more actuators can also be coupled to user-interface 
112, for example, to convey a virtual touch such to a user. In yet another embodiment, a 
plurality of actuators can be coupled to housing 110 as well as user-interface 112. In 
addition, one or more actuators may also be coupled to a headset, a wristband, or other 

20 accessory means associated Avith a handheld communication device. 
Embodiments of the invention include the following. 

In one embodiment, an individual (or 'TBob") can have a mobile phone according to 
the invention. The mobile phone also has an e-mail capability, for example, including both 
"receive" and "send"). The mobile phone is configured to provide a pluraUty of haptic 

25 effects, e.g., by including appropriate hardware (such as actuators described above) and 
program code. Bob can program the mobile phone, for example, via user-interface 112 
through API 150 shown in FIG. 1, by inputting various events of interest and associating each 
with a distinct haptic effect. Thus, when an event of interest is subsequently received, the 
mobile phone provides tlie corresponding haptic effect. 

30 In one embodiment, Bob's phone includes progi*anmiing that provides a first haptic 

effect when an input signal is received fi'om the mobile phone of Bob's wife (or "Alice"). 
Bob's phone also includes programming that j^rovides a second haptic effect that is different 
and distinct jfrom the first haptic effect, when an input signal is received from the mobile 
phone of Bob's supervisor at work (termed "Carol" herein). Bob's phone is further be 
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configured to provide a third haptic effect that is different Jfrom the two mentioned above, 
e.g., when an e-mail is received fi-om the e-mail address of Bob's stock broker (where the e- 
mail contains a "smiley-face" emoticon, for instance). The third haptic effect can be a 
vibration with high magnitude and short duration, e.g., to emulate a "high-five." 

hi another embodiment. Bob can be watcliing a movie in a theater with liis mobile 
phone in his pocket. It is set to malce no noise, because Bob is in a theater. While Bob is 
watching the movie, Bob's mobile phone vibrates with the second haptic effect mentioned 
above. Bob chooses to ignore the call, because he does not wish to speak witli his supervisor 
at a movie theater. Later, Bob's mobile phone vibrates with the first haptic effect. Bob wants 
to speak with Alice, for example, to make plans to meet later. So Bob answers the phone and 
quickly exits the theater to talk with Alice. 

Bob's mobile phone can also include a personal schedule/calendar application. After 
speaking v^dth Alice, Bob can enter an entry in the calendar at the 7:00PM tinae mark — "Meet 
Alice". Bob can also choose a fourth haptic effect to associate with the calendar entry. The 
mobile phone can be programmed to output tli^'fourtli haptic effect fifteen minutes before the 
time entiy of the calendar (i.e., at 6:45PM). 

Bob's mobile phone can be equipped with GPS capability, along with an associated 
application program for location determination. Bob can also store addresses of various 
locations of interest in the application program, hi one embodiment. Bob can be on the l oad. 
Bob's mobile phone vibrates with a distinct fifth haptic effect. Bob recognizes the fifUi 
haptic effect being associated with the haptic logo of his favorite electronics store. He then 
checks with tlie apphcation program, and receives a sixth haptic effect associated with the 
distance between his current position and tlie store location. Bob then decides to make a stop 
at the store. 

A haptically-enabled handheld communication device of the invention may be further 
used as a two-way haptic remote control, for example, for controlling a remote system such 
as a Television set or a multimedia system. In one embodiment, the events as referred to 
above may be related to program channels shown on the remote system, each identified by a 
channel number (which may be used as the "source"), for instance. The corresponding haptic 
effects may be customized on a per-chaimel basis. Such haptic effects can serve, to mfomi a 
user as to which channel is on, as a user is channel-surfing by way of this haptic remote 
control, so that the user need not to look up the display screen. 

FIG. 9 depicts a flowchart illustrating a method for providing haptic effects to a 
remote control in one embodiment of the presoit invention, hi the embodiment shown, the 
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remote control sends a command signal to a remote system at step 910. As with a 
conventional remote control, the signal may or may not reach the television. The remote 
control then determines whether a feedback signal has been received at step 920. If tlie 
remote control receives a feedback signal; the remote control provides a first haptic effect at 
step 930. If not, tiie remote control provides a second haptic effect at step 940. 

The first haptic effect can be fiuther customized according to the received feedback 
signal. In one embodiment, the remote system provides information (e.g., via the feedback 
signal) to the remote control regarding the state of the display, e.g., based on a predetermined 
scheme. The remote control may use the information to determine a corresponding haptic 
effect to provide at step 930. In alternative embodiments, the remote system may determine 
the appropriate haptic effect to provide and include a corresponding haptic code in the 
feedback signal. Tlie remote control provides the haptic effect at step 930 based on this 
haptic code. 

The foregoing description of the preferred embodiments of the invention has been 
presented only for the purpose of illustration and description and is not intended to be 
exhaustive or to limit the mvention to the precise forms disclosed. Numerous modifications 
and adaptations thereof will be apparent to those slciUed in the art without departing from the 
spirit and scope of the present invention. 
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That wliich is claimed is: 

1. A metliod, comprising: 

receiving an input signal associated with a virtual touch; 

outputting a request relating to a contact with a user-interface member coupled to a 
handheld communication device; and 

providing a control signal associated with the contact to an actuator coupled to the 
handheld communication device, the control signal configured to cause the actuator to output 
a haptic effect associated with the virtual touch. 

2. The method of claim 1 further comprising extracting a haptic code fi-om the input 
signal, the control signal being based at least in part on the haptic code. 

3. Tlie method of claim 1 wherein the user-interface member includes one of a key, a 
button, a key pad, a direction pad, a touch screen, a scroU wheel, a mini-joystick, a trackball, 
and a knob. 

4. Tlie method of claim 1 wherein the virtual touch is associated with one of a 
handshake, a high-five, a pat on the back, a pulse sensation, a heartbeat sensation, and a pet 
purring sensation. 

5. A method, comprising: 
receiving a virtual touch indicator, 

performing an initialization responsive to the virtual touch indicator on a handheld 

commimication device; 

receiving a virtual touch signal associated with the initialization; and 

outputting a control signal associated with the virtual touch signal to an actuator 

coupled to tlae handheld communication device. 

6. The method of claim 5 wherein the actuator is configured to output a haptic effect to a 
user-interface member coupled to the handheld communication device. 

7. Tlie method of claim 6 wherein tlie user-interface member includes one of a key, a 
button, a key pad, a direction pad, a touch screen, a scroll wheel, a mini-joystick, a trackball, 
and a knob. 

8. The method of claim 5 wherein the initialization includes outputting a request relating 
to a contact with the user-interface member. 

9. The method of claim 5 wherein the virtual touch signal is associated with a 
manipulation of a remote user-interface member. 

10. A computer-readable medium on which is encoded program code, comprising: 
program code for receiving an input signal associated with a vutual touch; 
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program code for outputting a request relating to a contact with a user-interface 
member coupled to a handheld communication device; and 

progiam code for providing a control signal associated with the contact to an actuator 
coupled to the handheld communication device, the control signal configured to cause the 
actuator to output a haptic effect associated with virtual touch. 

11. The computer-readable medium of claim 10 further comprising extracting a haptic 
code from the input signal, the control signal being based at least in part on the haptic code. 

12. The computer-readable medium of claim 10 wherein the virtual touch is associated 
with one of a handshake, a high-five, a pat on the back, a pulse sensation, a heartbeat 
sensation^ and a pet purring sensation.. 

13. A computer-readable medium on which is encoded program code, comprising: 
program code for receiving a virtual touch iadicator; 

program code for performing an initialization responsive to the virtual touch indicator 
on a handheld communication device; 

program code for receiving a virtual touch signal associated with the initialization; 

and 

program code for outputting a control signal associated with the virtual touch signal to 
an actuator. 

14. The computer-readable medium of claim 13 wherein the actuator is configured to 
output a haptic effect to a user-interface member coupled to the handheld communication 
device. 

15. The computer-readable medium of claim 14 wherein the user-interface member 
includes one of a key, a button, a key pad, a direction pad, a touch screen, a scroll wheel, a 
mini-joystick, a trackball, and a knob. 

16. The computer-readable medium of claim 13 wherein the initialization includes 
outputting a request relating to a contact with the user-interface member. 

17. A data stream embodied in a carrier signal, carrying instructions to: 
receive an input signal associated with a virtual touch; 

output a request relating to a contact with a user-interface member coupled to a 
handheld communication device; and 

provide a control signal associated witli the contact to an actuator coupled to the 
handheld communication device, the control signal configured to cause the actuator to output 
a haptic effect associated vnih tlie virtual touch. 
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18. A data stream embodied in a carrier signal, carrying instructions to: 
receive a virtual touch indicator; 

perfomi an initialization responsive to the virtual touch indicator on a handheld 

communication device; 

receive a virtual touch signal associated with the initiahzation; and 

output a control signal associated with the virtual touch signal to an actuator coupled 

to the handheld commmiication device. 

19. The apparatus, comprising: 

a user-interface member coupled to a body; 
aprocessoi^ 

an actuator coupled to the body and in communication with the processor; and 
a memory in communication with the processor, the memory storing program code 
executable by the processor, including: 

program code for receiving an input signal associated with a virtual touch; 

program code for outputting a request relating to a contact with the user- 
interface member; and 

program code for providing a control signal associated with the contact to the 
actuator, the control signal configured to cause the actuator to output a haptic effect 
associated with the virtual touch. 

20. The apparatus of claim 19 wherein the body is included in a handheld communication 
device. 

21. The apparatus of claim 20 wherein the handheld communication device includes one 
of a cellular phone, a satellite phone, a cordless phone, a personal digital assistant, a pager, a 
two-way radio, a portable computer, a game console controller, a personal gaming device, 
and an MPS player. 

22. The apparatus of claim 20 wherein the user-interface member incKides at least one of 
a key, a button, a key pad, a direction pad, a touch screen, a scroll wheel, a mini-joystick, a 
trackball, and a Icnob. 

23. The apparatus of claim 19 wherein the virtual touch is associated with one of a 
handshalce, a high-five, a pat on tlie back, a pulse sensation, a heartbeat sensation, and a pet 
purring sensation. 

24. The apparatus, comprising: 
a user-interface member; 

a processor; 
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an actuator coupled to the a user-interface member and in commurdcation with the 

processor; and 

a memory in communication with the processor, the memory storing program code 
executable by the processor, including: 

program code for receiving a virtual touch indicator; 

program code for performing an initialization responsive to the vutual touch 
indicator; 

program code for receiving a virtual touch signal associated with the 
initialization; and 

program code for outputting a control signal associated with the virtual touch 
signal to the actuator. 

25. The apparatus of claim 24 wherein .the user-interface member is coupled to a 
handheld commimication device. 

26. Tlie apparatus of claim 25 wherein the handlield communication device includes one 
of a cellular phone, a satellite phone, a cordless phone, a personal digital assistant, a pager, a 
two-way radio, a portable computer, a game console controller, a personal gaming device, 
and an MPS player. 

27. The apparatus of claim 24 wherein the user-interface member includes at least one of 
a key, a button, a key pad, a direction pad, a touch screm, a scroll wheel, a mini-joystick, a 
trackball, and a knob. 

28. The apparatus of claim 24 wherein the virtual touch signal is associated with a 
manipulation of a remote user-interface member. 
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24 June 2004 (24.06.2004) 



Applicant's or agent's file reference 
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1. Notice is hereby given that the International Bureau has communicated, as provided in Article 20, the international application to the 
following designated Offices on the dale indicated above as the dale of mailing of this notice: 

AU. AZ. BY. CH. CN, CO, DZ, EP, HU, JP, KG, KP. KR, MD. MK, MZ, RU, TM. US 

In accordance with Rule 47.1(c), third sentence, those Offices will accept the present notice as conclusive evidence that the communication 
of the international application has duly taken place on the date of mailing indicated above and no copy of the international application is 
required to be furnished by the applicant to the designated Office(s). 

2. The following designated Offices have waived the requirement for such a communication at this time: 

AE AG AL, AM, AP, AT, BA, BB, BG, BR. BZ, CA, CR. CU. CZ, DE, DK. DM, EA, EC. EE, EG. ES. Fl. GB, GD. GE, 
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PL. PT, RO, SC. SD. SE. SG. SK. SL. SY, TJ. TN. TR. TT, TZ, UA. UG. UZ. VC, VN. YU. ZA. ZM, ZW 

The communication will be made to those Offices only upon their request Furthermore, those Offices do not require the applicant to 
fumish a copy of the international application (Rule 49. liBrbis)). 

3. Enclosed with this notice is a copy of the international application as published by the International Bureau on 
24 June 2004 (24.06,2004) under No. WO 2004/053829 

4. TIME LIMITS for filing a demand for international preliminary examination and for entry into the national phase 

The applicable lime limit for entering the national phase will, subject to what is said in the foUowing paragraph, be 30 MONTHS from 
the priority date, not only in respect of any elected Office if a demand for international preliminary examination is filed before the 
expiration of 19 months from the priority date, but also in respect of any designated Office, in the absence of filing of such demand, where 
Article 22(1) as modified with effect from 1 April 2002 applies in respect of that designated Office. For further details, see PCT Gazette 
No. 44/2001 of 1 November 2001, pages 19926, 19932 and 19934, as well as the PCT Newsletter, October and November 2001 and 
February 2002 issues. 

In practice, time limits other than the 30-month time Dmit will continue to apply, for various periods of time, in respect of certain 
designated or elected Offices. For regular updates on the applicable time limits (20, 2 1 , 30 or 3 1 months, or other time limit). Office by 
Office, refer to the PCT Gazette, the PCT Newsletter and the PCT Applicant's Guide, Volume 11, National Chapters, all available firom 
WlPo' s Internet site, at http://www.wipo.int/pct/en/index.html. ^ ^^^^1 

For filing a demand for international preliminary examination, sedTBfe IX. Only an 

applicant who is a national or resident of a PCT Contracting Stale which jggboAil. by cgiapter. n. hl& jiCj ht to file a demand for 
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international preliminary examination (at present, all PCT Contracting States; afe. bound by Chapter 1 1). 

It is the applicant's sole responsibility to monitor all these time limits. 
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Whoi? tht time linut for filing such amendments is nonnally two months from the date of transmittal of the 
international seardi rqx>rt. 

Whffe? Wrectly to the International Bureau of WIPO, 34, chemin des Cdonibettes 
121 1 Gei»va 20, Switzerland, Facsimile No.: (41-22) 740.14.35 

For more detailed instructions, see the notes on the accompanying sheet. ' 

The applicant is herein notified that no international seareh report will be established Mid that the declaration under 
Article I7(2)(a) to that effect is transmiaed herewith. 

With regard to the protest against payment of (an) additional fee(s) under Rule 40.2. the applicant is notified that: 

Q the protest together with the decision thereon has been transmitted to the International Bureau together with the 

applicant's request to forward the texts of bodi the protest and the decision thereon to the designated Offices. 
I I no dedsion has been made yet on the protest; the applicant will be notified as soon as a decision is made. 

4. Reminders 

Shortly after 18 months from the priority date, the international application will be published by the IntemaUonal Bureau. If the 
aoDlicant wishes to avoid or pos^xwie publication, a notice of withdrawal of the international application, or of the priority claim, must 
reach the International Bureau as provided in Rules 90 W5.I and 90 «r.3, respectively, before the completion of the teduiscal 
preparations for international publication. 

Within 19 m<Miths from the priority date, but only in respect of some designated Offices, a demand for international prelinunary 
examination must be filed if tiie applicant wishes to pos^>one the entjy into the national phase until 30 months from the priority date 
Cm some Offices even later); otiierwise the applicant must, within 20 months from tiie priority date, perform the prescribed acts for 
entiy into the national ]diase before those dengnated Offices. 

In respect of other designated Offices, the time limit of 30 months (or later) will apply even if no demand is filed within 19 months. 
See the Annex to Form PCT/IB/301 and, for details about the applicable time linuts. Office by Office, s^ tiie PCT Applicani's Guide, 
Vdume II, National Ch^ners and die WIPO Internet site. 
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This international search report consists of a total of . ^ sheets. 

It is also accompanied by a copy of each prior art document cited in this report. 
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Basis of the Report 

a. With regard to the language, the international search was carried out on the basis of the international application in the 

language in v^^ch it was filed, unless otherwise indicated under this item. 
I I the intemational search was carried out on the basis of a translation of die international fq^plication furnished to this 
Authority (Rule 23.1(b)). 

With regard to any nucleotide and/or amino acid sequence disclosed in die intemational application, the intemational 
search was carried out on the basis of the sequence listing: 

contained in the international application in written form. 

filed together with the intemational application in concqputer readable form. 

furnished subsequently to this Authority in written form. 

furnished subsequently to this Authority in computer readable form. 

the statement that the subsequendy furnished written sequence listing does not go beyond the disclosure in the 
international application as filed has been furnished. 

the statement that the information recorded in computer readable form is identical to the written sequence listing has 
been furnished. 

Certiun claims were found unsearchable (See Box I). 

Unity of invention is lacking (See Box 11). 
With regard to die tHle, 

^1 the text is approved as submitted by the qjplicant. 
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may, within one month from the date of mailing of this intemational search report, submit comments to this 
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□ 
□ 
□ 
□ 
□ 

□ 

□ 
□ 



6. 



The figure of the drawings to be published with the abstract is Figure No. 1, 

CXj as suggested by the applicant. 

I I because the applicant failed to suggest a figure. 

I I because this figure better characterizes the invention. 



□ 



None of the figures 



Form PCT/lSA/210 (first sheet) (July 1998) 



INTERNATIONAL SEARCH REPORT 



InteTnational application No. 

PCT/US03/38862 



Box III TEXT OF THE ABSTRACT (Continuation of Item 5 of the First sheet) 



Embodiments of the invention relate to methods and systems (100 of figure 1) for providing customized "haptic messaging" to use 
of handheld communication devices in a variety of ^plications. In one embodiment, a method of providing virtual touch to a 
handheld communication device includes: receiving an ii^ut signal associated with a virtual touch; outputing a request relating to a 
contact with a user-interface member coupled to a handheld communication device; and providing a control signal associated with 
the contact to an actuator coupled to the handheld communication device, the control sig^ beii^ configured to cause ttie actuator 
to ouqput a haptic effect associated with die virtual touch. 



Form PCT/ISA/210 (continuation of fu^st sheet(2)) (July 1998) 




INTERNATIONAJL SEARCH REPORT 


Internationa] application No. 




PCT/US03/38862 


A. CLASSIFICATION OF SUBJECT MATTER 




IPC(7) : G09G 5/00 




US CI. : 345/156 




According to IntermtionaS Patent Oassiiication (TPC^ or to both national classification and IPC 


B. FIELDS SEARCHED 



Minimum documentation searched (classiiication system followed by dassiHcatiOQ symbols) 
U.S. : 345/156, 157,163,169,173,179.184; 441/5,730; 340/825.46,825.19; 709/203,217 



Documentation searched other than minifrmm dociunentation to the extent that such documents are included in the fields searched 



Electronic data base consulted during the intemational search (name of data base and, vAiiere practicable, search terms used) 



DOCUMENTS CONSIDERED TO BE REaUEVANT 



Category * 


Citation of document, with indication, >^ere appropriate, of the relevant passages 


Rcdevant to claim No. 


y 


US 6,429,846 B2 (Rosenberg et al.) 06 August 2002 (06.08.2002) Col. 1 , lines 23-67, 
Col.2, Lines 6-53. Col. 3, Lines 39-62, Col. 4, Lines 65-^7, Col. 5, Lines 32-55, Col. 6, 
Lines 7-49. 55-67, Col. 8, Lines 61-67. Col. 11, Lines 24-39, Col. 13, Lmes 22-40, 27- 
41, Col. 14. Lines 54-67, Col. 15, lines 45-52, Col. 16, lines.8-34. Col. 17, Lines 1-13, 
18-67, Col. 18, Lmes 10-67, Col. 20, Lines 1-18, 23-27 


1-28 


Y 


US 6,337,678 81 (Hsh) 08 January 2002 (08.01 .2002) C<d. 17, Lines 43-63 


1-28 


y 


US 6.219,032 Bl (Rosenberg et al.) 17 April 2001 (17.04.2001) Col. 7, Lines 1-66, Col. 
8, Lines 1-67, Col. 19, Line 59 to Col. 20, Linel6 


1-28 


y 


US 6.435.794 Bl (^ringer) 20 August 2002 (20.08.2002) Col. 10, Lines 20-67 


1-28 



I I Further documents are listed in the continuation of Box C. | | See patent family annex. 



* Speda] categories <if c&ed documoiis: 

"A" document denning the general state of Ibe ait which is not cflnsidered to he 
of particular relevance 

'E" earlier appficaiicn txr patent published cd or afio' the fatematicDa] fUfaig date 

"L* document whicb may ifarow doubts on priority ckum(s) or which , is dted to 
esiablbh Ibe publtcation date of another citation or other spedal reason (as 
spedfiecQ 

*0* document referring to an oral disclosure, use, ejdiibilioa or other means 

"P" document pubCsbed prior to the imemaiioDal filbig date but later than Ibe 
priority date claimed 



later document published after the intematioaa] filing date or priority 
date and not m cooflfet witli the application but cited to understand the 
prindple or Ibeoor und6ri>riiig liie inveaiioo 

document of particular relevance; tbe daSmed inwentioo cannot be 
considered novel or cannot be coosidered to involve an inventive stqi 
when the docimient is talceo alone 

documeoi of particular relevance; the claimed inventiaa cannot be 
considered to involve an inventive step when tbe doounem is 
combiDcd with ooe or more other such documents, such corobinaiion 
being obvious to a pereoo sldUed in the an 

document member of Ibe saone patent fanuly 



Date of the actual con9)letion of the international search 
25 March 2004 f25.03.2004) 


Date of mailing of the^p^n^^c^^ 




Name and mailing address of the ISA/US 
Mail Stop per. Atui: ISA/US 
CommissioDer for Patents 

P.O. Box 1450 C 
Alexaodria. Viiginia 22313-1450 
facsimile No. (703)305-3230 


Apl^^jzedJ3fficer y^y^,^,..^^ 
/^Prabodh Dharia // o 
Telephone No. 703-605-1231 





CO 

I 

Q 

a; 

CD 

O 
o 



Fonn FCT/ISA/210 (second sheet) (July 1998) 




wu zuu^/ud^dzy ai iiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiiuiiiiiiiiuiiiiiiiiiiiiiiiiijiiniiiuiiiiiiiflm 



before the expiration of the time limit for amending the For two-letter codes and other abbreviations, refer to the "Guid- 
claims and to be republished in the event of receipt of ance Notes on Codes and Abbreviations-' appearing at the begin- 
amendments ning of each regular issue of the PCT Gazette, 



INTERNATIONAL SEARCH REPORT 



International application No. 
PCT/US03/38862 



^ CLASSIFICATION OF SUBJECT MATTER 

IPC(7) : G09G 5/00 

USCL : 345/156 . 
According to inti>rn«tional Patent aassifi c ation (IPC) or to both national classification and IPC 
B. FIELDS SEARCHED ■ 



Minimum documentation searched (classification system followed by classification qrmbols) 
U.S. : 345/156,157,163,169,173,179,184; 441/5,730; 340/825,46,825.19; 709/203,217 



Documentation 



searched other than minimum documentation to the extent that such documents are included in die fields searched 



Electronic 



data base consulted during the international search (name of data base and. vAiere practicable, search terms used) 



-Q DOCUMENTS CONSIDERED TO BE RELEVANT 



Category 



Y 

y 
y 



Citation of document, with uadication, where appropriate, of the relevant passages 



US 6 429,846 B2 (Rosenberg et al.) 06 August 2002 (06.08.2002) Col. 1 , lines 23-67, 
Col 2 Lines 6-53. Col. 3, Lines 39-62, Col. 4, Lines 65-67, Col. 5, Lines 32-55, Col. 6, 
Lines 7-49, 55-67. Col. 8, Lines 61-67, Col. 11. Lines 24-39, Col. 13, Lines 22-40, 27- 
41, Col. 14. Lines 54-67, Col. 15. lines 45-52, Col. 16. lines 8-34. Col. 17, Lines 1-13. 
18^7, Col. 18, Lines 10-67. Col. 20, Lines 1-18, 23-27 

US 6,337,678 HI (Fish) 08 January 2002 (08.01.2002) Col. 17, Lines 43-63 

US 6,219,032 Bl (Rosenberg et al.) 17 >^rU 2001 (17.04.2001) Col. 7, Lines 1-66, Col. 
8, Lines 1-67. Col. 19, Line 59 to Col. 20, Unel6 

US 6.435.794 Bl (Springer) 20 August 2002 (20.08.2002) Col. 10, Lines 20-67 



Relevant to claim No. 



1-28 



1-28 
1-28 
1-28 



|~| Further documents are listed m the continuation of Box C. Q See patent family annex 



"L" 



Special categories of cited documents: 

document denning the genera] state of the ait which U n« ccnsidered to be 
of particular refevance 

earlier appBcaiioo or patent published qq or after the intematiaaa) filing date 

document which may throw doubts on priority cUum($) or which is cited to 
establish the publication date of another citatioo or other special reascn (as 
specirieiO 

document referring lo an oral disclosure, use. exhibition or «her means 

document published prior lo the mteniatiooal nimg date but later than the 
priority dale claimed 



-X- 



later document published after the interoational filing date or priority 
date and not to cooflict with the appUcatioa but cited to understand the 
principle or theory underiying the mveotioD 

document of particular relevance; die claimed invention cannot be 
coosidei«d novel or cannot be considered to involve an inventive step 
when the dociment is taken alone 

document of particular relevance; (he claimed invention cannot be 
considered to involve an inventive step when the document is 
combined with one or more other such documenu, such combination 
being obvious to a person skilled in the art 

document member of the same patent family 



Date of the actual completion of the international search 
25 March 2004 (25.03.2004) 



Name and mailing address of the ISAAJS 
Mail Slop POT, Attn: ISA/US 
Commissiooer for Patents 
P.O. Box 1450 

Alexandria. Viiginia 22313-1450 
Facsimile No. (703)305-3230 



Date of mailing of the international search report 

12 APR 2001 



>^^or)2ed.^ficer 
/^abodh Dharia 



Telephone No. 703-605-1231 




Form PCT/lSA/210 (second sheet) (July 1998) 




NOTES TO FORM PCT/ISA/220 



These Notes are intended to give the basic instructions concerning the filing of amendments under Article 19. The 
Notes are based on the requirements of the Patent Cooperation Treaty, the Regulations and the Administrative 
Instructions under that Treaty. In case of discrepancy between these Notes and those requirements, the latter are 
applicable. For more detailed information, see also the PCT Applicant 's Guide^ a publication of WIPO. 

In these Notes, "Article," "Rule" and "Section" refer to the provisions of the PCT, the PCT Regulations and the PCT 
Administrative Instructions, respectively. 



INSTRUCTIONS CONCEre^^^^^GAJVlENDME^^^ 19 



The applicant has, after having received the international search report, one opportunity to amend the claims of the 
international application. It should however be emphasized that, since all parts of the intemational application (claims, 
description and drawings) may be amended during the intemational preliminary examination procedure, there is usually 
no need to file amendments of the claims under Article 19 except where, e.g. the applicant wants the latter to be published 
for the purposes of provisional protection or has another reason for amending the claims before intemational publication. 
Furthermore, it should be emphasized that provisional protection is available in some States only. 

What parts of the international application may be amended ? 

Under Article 19, only the claims may be amended. 

During the intemational phase, the claims may also be amended (or further amended) under Article 34 before 
the Intemational Preliminary Examining Authority. The description and drawings may only be amended 
under Article 34 before the Intemational Preliminary Examining Authority. 

Upon entry into the national phase, all parts of the intemational application may be amended under Article 28 
or, where applicable. Article 41. 



When ? Within 2 months from the date of transmittal of the intemational search report or 16 months from the priority 
date, whichever time limit expires later. It should be noted, however, that the amendments will be considered 
as having been received on time if they are received by the Intemational Bureau after the expiration of the 
applicable time limit but before the completion of the technical preparations for intemational publication 
(Rule 46.1). 



Where not to file the amendments ? 

The amendments may only be filed with the Intemational Bureau and not with the receiving Office or the 
Intemational Searching Authority (Rule 46.2). 

Where a demand for intemational preliminary examination has been/is filed, see below. 



How? Either by cancelling one or more entire claims, by adding one or more new claims or by amending the text of 
one or more of the claims as filed. 

A replacement sheet must be submitted for each sheet of the claims which, on account of an amendment or 
amendments, differs from the sheet originally filed. 

All the claims appearing on a replacement sheet must be numbered in Arabic numerals. Where a claim is 
cancelled, no renumbering of the other claims is required. In all cases where claims are renumbered, they must 
be renumbered consecutively (Administrative Instructions, Section 205(b)). 

The amendments must be made in the language in which the international application is to be published. 



What documents must/may accompany the amendments ? 
Letter (Section 205(b)): 

The amendments must be submitted with a letter. 

The letter will not be published with the intemational application and the amended claims. It should not be 
confused with the "Statement under Article 19(1)" (see below, under "Statement under Article 19(1)")- 
The letter must be In English or French, at the choice of the applicant. However, if the language of the 
international application is English, the letter must be in English; if the language of the international 
application is French, the letter must be in French. 



Notes to Form PCT/ISA/220 (first sheet) (July 1998; reprint April 2002) 



From the INTERNATIONAL BUREAU 



PCT 

NOTIFICATION CONCERNING 
SUBMISSION OR TRANSMITTAL 
OF PRIORITY DOCUMENT 

\ru 1 AOrninisiraiive insiruciions, oeciion 4i i; 


To: 

ALEMANNI, John, C. 
Kilpatrick Stockton LLP 
1 001 West Fourth St. 

Win^tnn-^alpm NP 97101 

United States of America 


04 March 2004 (04.03.2004) 




Applicant s or dQcni s iiie reierence 
IMM152D.PCT 


IMPORTANT NOTIFICATION 


international application No. 
PCT/US2003/038862 


International filing date (day/month/year) 
08 December 2003 (08.12.2003) 


International publication date (day/month/year) 
Not yet published 


Priority date (day/month/year) 

08 December 2002 (08.12.2002) 


Applicant 

IMMERSION CORPORATION eta! 





1. By means of this Form, which replaces any previously issued notification concerning submission or transmittal of priority 
documents, the applicant is hereby notified of the date of receipt by the International Bureau of the priority document(s) 
relating to all earlier dpplication(s) whose priority is claimed. Unless otherwise indicated by the letters •'NR", in the 
right-hand column or by an asterisk appearing next to a date of receipt, the priority document concerned was submitted or 
transmitted to the International Bureau in compliance with Rule 17.1(a) or (b). 



2. (If applicable) The letters "NR" appearing in the right-hand column denote a priority document which , on the date of mailing 
of this Form, had not yet been receh^ed by the International Bureau under Rule 17.1 (a) or (b). Where, under Rule 1 7.1 (a) , the 
priority document must be submitted by the applicant to the receiving Office or the International Bureau, but the applicant 
fails to submit the priority document within the applicable time limit under that Rule, the attention of the applicant is directed 
to Rule 17.1(c) which provides that no designated Office may disregard the priority claim concerned before giving the 
applicant an opportunity, upon entry into the national phase, to furnish the priority document within a time limit which is 
reasonable under the circumstances. 



(If applicable) An asterislc(*) appearing next to a date of receipt, in the right-hand column, denotes a priority document 
submitted or transmitted to the International Bureau but not in compliance with Rule 17.1(a) or (b) (the priority document 
was received after the time limit prescribed in Rule 17.1(a) or the request to prepare and transmit the priority document was 
submitted to the receiving Office after the applicable time limit under Rule 17.1(b)). Even though the priority document was 
not furnished in compliance with Rule 17.1(a) or (b), the International Bureau will nevertheless transmit a copy of the 
document to the designated Offices, for their consideration. In case such a copy is not accepted by the designated Office 
as priority document. Rule 17.1(c) provides that no designated Office may disregard the priority claim concerned before 
giving the applicant an opportunity, upon entry into the national phase, to furnish the priority document within a time limK 
which is reasonable under the circumstances. 



Priority ^ptg 
08 Dece 2002 (08.12.2002) 



Priority application No. 



Country orlhBgiqhaj Ofrige 
or PCT receiving 6H\ce 



of priority document 



60/431,662 



MAR 1 5 200^ 



,>.. rst-^r>ipt/ .^^^-r^>A!/•r 



01 Marc 2004 (01 .03.2004) fd. 

b 

IT 

n 



1 



The Intemationai Bureau of WlPO 
34, chemin des Colombettes 
121 1 Geneva 20, Switzeriand 



Facsimile No. (41-22)338.87.40 



Authorized officer 

N. BEN MANSOUR 

Telephone No. (41-22)338 8761 



T3 



Form PCT/IB/304 (January 2004) 



006134008 



PCT REQUEST 



Original (for SUBMISSION) - printed on 08.12.2003 01:01:46 PM 



IMM152D.PCT 



0 


For receiving Office use only 




0-1 


International Application No. 




0-2 


International Filing Date 




0-3 


Name of receiving Office and "PCT 
International Application" 






0-4 


Form - PCT/RO/101 PCT Request 




0-4-1 


Prepared using 


PCT-EASY Version 2 . 92 
(updated 01.07.2003) 


0-5 


Petition 

The undersigned requests that the 
present international application be 

prOCeSSeO accoiumy lu hic naiciu 

Cooperation Treaty 




0-6 


Receiving Office (specified by the 


United States Patent and Trademark 




applicant) 


Office (USPTO) (RO/US) 


0-7 


Appiicant's or agent's file reference 


IMM152D.PCT 


1 


Title of invention 


METHODS AND SYSTEMS FOR PROVIDING A 
VIRTUAL TOUCH HAPTIC EFFECT TO HANDHEIjD 
COMMUNICATION DEVICES 


11 


Applicant 




n-1 


This person is: 


applicant only 


11-2 


Applicant for . 


all designated States except US 


11-4 


Name 




11-5 


Address: 


801 Fox Lane 

San iJose, CA 95131 

United States of America 


11-6 


State of nationality 


US 


11-7 


State of residence 


US 


IIM 


Applicant and/or inventor 




111-1-1 


This person is: 


applicant and inventor 


!ll-1-2 


Applicant for 


US only 


IIM-4 


Name (LAST. First) 


GRANT ^ Danny 


HI-1-5 


Address: 


6678 Chateaubriand 
Montreal, Quebec H2S 2N7 
Canada 


III-1-6 


State of nationality 


CA 


III-1-7 


State of residence 


CA 



PCT REQUEST 



Originai (for SUBMISSION) - printed on 08.12.2003 01:01:48 PM 



IMM152D.PCT 



Applicant and/or inventor 
This person is: 
Applicant for 
Name (LAST. First) 
Address: 



State of nationality 
State of residence 



applicant and inventor 

US only 

EID , Jeffrey 

105 Shadwell Drive 

Danville, CA 94506 

United States of America 

US 

US 



Applicant and/or inventor 
This person is: 
Applicant for 
Name (LAST. First) 
Address: 



State of nationality 
State of residence 



applicant and inventor 

US only 

ENDO , Shoichi 

202 68 Northwest Square 

Cupertino, CA 95014 

United States of America 

JP 

US 



Agent or common representative; or 
address for correspondence 
The person identified t>elow is 
hereby/has been appointed to act on 
behalf of the applicant(s) before the 
competent Intemationai Authorities as: 
Name (LAST. First) 

Address: 



Telephone No. 
Facsimile No. 
e-mail 

Agent's registration No. 



agent 

ALEMANNI , John , C . 
Kilpatrick Stockton IiIiP 
1001 West Fourth St. 
Winston-Salem, NC 27101 
United States of America 
336-607-7300 
336-607-7500 

jalemanni@kilpatrickstockton . com 
47,384 



Additional agent(s) 



Name(s) 



additional agent (s) with same address as 
first named agent 

STOJKOVICH, Goran, P. (45,841); WINTERLE, 

Bret, T. (54,546); GARDNER, 

S teven (41,772); HARRINGTON , John , 

M. (25,592) 



PCT REQUEST 



Original (for SUBMISSION) - printed on 08.12.2003 01:01:48 PM 



IMM152D.PCT 



Designation of States 



Regional Patent 

(other kinds of protection or treatment, 
if any. are specified between 
parentheses after the designatton(s) 
concerned) 



AP: 6H GM KE LS MW MZ SD SL SZ TZ UG ZM 
ZW and any other St:at:e which is a 
Contracting State of the Harare Protocol 
and of the PCT 

EA: AM AZ BY KG KZ MD RU TJ TM and any 
other State which is a Contracting State 
of the Eurasian Patent Convention and of 
the PCT 

EP: AT BE BG CH&LI CY CZ DE DK EE ES FI 
FR GB GR HU IE IT LU MC NL PT RO SE SI 
SK TR and any other State which is a 
Contracting State of the European Patent 
Convention and of the PCT 

OA: BF BJ CF CG CI CM GA GN GQ GW ML MR 
NE SN TD TG and any other State which is 
a member State of OAPI and a Contracting 
State of the PCT 



National Patent 

(other kinds of protection or treatnnent. 
if any. are specified between 
parentheses after the designation(s) 
concerned) 



A£ AG Ali 

CA CH&LI 

EE EG ES 

IN IS JP 

LU LV MA 

OM PG PH 

SY TJ TM 

YU ZA ZM 



AM AT 

CN CO 

FI GB 

KE KG 

MD MG 

PL PT 

TN TR 
ZW 



AU AZ 

CR CU 

GD GE 

KP KR 

MK MN 

RO RU 

TT TZ 



BA BB BG 

CZ DE DK 

GH GM HR 

KZ LC LK 

MW MX MZ 

SC SD SE 

UA UG US 



BR BY BZ 

DM DZ EC 

HU ID IL 

LR LS LT 

NI NO NZ 

SG SK SL 

UZ VC VN 



Precautionary Designation Statement 

In addition to the designations made 
under Items V-1 , V-2 and V-3. the 
applicant also makes under Rule 4,9(b) 
all designations which would be 
permitted under the PCT except any 
designation(s) of the State(s) indicated 
under item V-6 below. The applicant 
declares that those additional 
designations are subject to confirmation 
and that any designation which Is not 
confimied before the expiration of 15 
months from the priority date is to be 
regarded as withdrawn by the applicant 
at the expiration of that time limit. 



Exclu5lon(s) from precautionary 
designations 



NONE 



Priority claim of earlier national 
application 

Filing date 

Number 
Country 



08 December 2002 (08.12.2002) 

60/431,662 

US 



PCT REQUEST 



Original (for SUBMISSION) - printed on 08.12.2003 01:01:48 PM 



IMM152D.PCT 



VI-2 


Priority document request 

The receiving Office is requested to 
prepare and transmit to the 
International Bureau a certified copy of 
the earlier application(s) identified 
above as item(s): 


VI-1 


VIM 


International Searching Authority 
Chosen 


United States Patent and Trademark 
Office (USPTO) (ISA/US) 


VIII 


Declarations 


- Number of dsclarations 




VIII- 1 


npriaration as to the identity of the 
inventor 






VIII-2 


Declaration as to the applicant's 
onfitic^mfsnt at the international filino 
date, to apply for and be granted a 
patent 






\/tii 1 


r^oHaratinn a<; to the aoolicant's 
entitlement, as at the international filing 
date, to claim the priority of the eariier 
application 






VMM 


Declaration of inventorship (only for the 
purposes of the designation of the 
United States of America) 






VIII-5 


Declaration as to non-prejudicial 
disclosures or exceptions to lack of 

novelty 






IX 


Check list 


number of sheets 


electronic file(s) attached 


IX-1 


Request (including declaration sheets) 


5 




IX-2 


Description 


14 




IX-3 


Claims 


4 




IX^ 


Abstract 


1 


EZABSTOO .TXT 


IX-5 


Drawings 


9 




1X-7 


TOTAL 


33 




Accompanying items 


paper document(s) attached 


electronic file(s) attached 


IX-8 


Fee calculation sheet 






lX-17 


PCT-EASY diskette 




Diskette 




Figure of the drawings which should 
accompany the abstract 


1 


IX-20 


Language of filing of the 
international application 




X-1 


Signature of applicant, agent or 
common representative 






X-1-1 


Name (LAST. First) 


^EMANNI, John, C. 





FOR RECEIVING OFFICE USE ONLY 



10-1 


Date of actual receipt of the 
purported international application 




10-2 


Drawings: 




10-2-1 


Received 




10-2-2 


Not received 




10-3 


Corrected date of actual receipt due 
to later but timely received papers or 
drawings completing the purported 
international application 





PCT REQUEST IMM152D.PCT 

Original (for SUBMISSION) - printed on 08.12.2003 01:01:48 PM 



10-4 


Date of timely receipt of the required 
corrections under PCT Article 11(2) 




10-5 


International Searching Authority 


ISA/US 


10-6 


Transmittal of search copy delayed 
until search fee is paid 




FOR INTERNATIONAL BUREAU USE ONLY 


11-1 


Date of receipt of the record copy by 
the International Bureau 





